Fine tuning CNNS with scarce training data - Adapting imagenet to art epoch classification

Deep Convolutional Neural Networks (CNN) have recently been shown to outperform previous state of the art approaches for image classification. Their success must in parts be attributed to the availability of large labeled training sets such as provided by the ImageNet benchmarking initiative. When training data is scarce, however, CNNs have proven to fail to learn descriptive features. Recent research shows that supervised pre-training on external data followed by domainspecific fine-tuning yields a significant performance boost when external data and target domain show similar visual characteristics. In this paper, we evaluate the performance of fine-tuned CNNs when the target domain is visually different from the dataset used to pre-train the model. Specifically, we address the problem of transfer learning from ImageNet domain to the task of classifying paintings into art epochs. Furthermore, we analyze the impact of training set sizes on CNNs with and without external data and compare the obtained models to linear models based on Improved Fisher Encodings. Our results underline the superior performance of fine-tuned CNNs but likewise propose Fisher Encodings in scenarios were training data is limited.